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What Preparation Works to Improve ACT 
Scores? 


Edgar I. Sanchez, PhD, and Raeal Moore, PhD 


Standardized tests like the ACT® provide a common basis for comparison of students 
across schools and states and have been a key point of consideration in postsecondary 
admissions for over 60 years. Research has shown that, in general, test preparation 

has a small but positive effect on standardized test scores. In this study, we focused 

on how three different test preparation formats—in-person instruction, self-paced, and 
other types—related to students’ ACT Composite scores. We examine whether there is a 
relationship between type of preparation activity and ACT Composite score and whether 
that relationship depends on family income or whether students are first-time or repeat 
ACT testers. For the total sample, we found no significant effects of test preparation on 
ACT Composite. We also found that the effects of the different preparation formats did 
not vary by family income. Furthermore, the number of times tested had an impact on 
ACT scores, and this effect differed by test preparation format. First-time test-takers who 
participated in self-paced test preparation had a lower ACT Composite score than those 
first-time test-takers who did not participate in a test preparation activity. This study 

does not align with past research that shows a small effect of test preparation on ACT 
scores. Our study might not have detected an effect because of the research design or 
the sample of students who participated in the survey. Or, heterogeneity in the quality, 
methodology, and modes of delivery within each test preparation format may have 
blurred any effects. 


Introduction 


Standardized tests like the ACT ensure a common basis for comparison of students 
across schools and states. This is one of the many reasons that the ACT has been such 
a key point of consideration in postsecondary admissions for over 60 years and is why 
many students take college entrance exams in an effort to maximize their postsecondary 
educational opportunities. In fact, among 2018 high school graduates, almost two million 
graduates took the ACT, and over 2.1 million graduates took the SAT. Of those taking the 
ACT, about 64% reported using some form of test preparation (e.g., study guides, online 
materials, practice tests, tutors, or test preparation courses).' Given the importance of 
test scores in college admissions decisions and scholarships, a multimillion dollar test 
preparation and tutoring industry has emerged (Barnes Report, 2017). 
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Test Preparation Effectiveness and Factors that Influence It 


Research has shown that, in general, test preparation has a small but positive effect on 
standardized test scores (Briggs, 2009; Moore, Sanchez, & San Pedro, 2018; Powers, 
1993; Schiel & Valiga, 2014a; Schiel & Valiga, 2014b). For example, the National 
Association for College Admissions Counseling (NACAC) reviewed efficacy studies 

on test preparation for the SAT and found minimal effects on test scores—an average 
gain of approximately 30 points (0.27 SD units) for subject tests and 40 points (0.20 SD 
units) for the total score (Briggs, 2009). For the ACT, Briggs (2001) found that students 
who participated in any type of test preparation scored a full point higher (0.17 SD 
units), on average, than those students who did not participate in test preparation. 


Delivery Format. While much of the research on test preparation effectiveness has 
focused on group instruction, recent research by Moore et al. (2018) showed that 
effects are stronger for students who worked with a private tutor or consultant relative 
to students who engaged in other test preparation activities that were either done 

on their own (e.g., online test preparation program) or in a group setting (e.g., test 
preparation workshops; see Bloom, 1984 and Ireson, 2004 for similar findings). To the 
authors’ knowledge, there is no prior research on whether self-paced test preparation 
compared to in-person instruction matters in improving students’ ACT or SAT scores. 
We will, therefore, explore this relationship. 


Retesting Effects. Test preparation that includes more practice opportunities, including 
re-testing, can have a positive effect on test score gains (Appelrouth & Zabrucky, 
2015). For students who retook a cognitive ability test, a meta-analysis found increases 
in test scores that were approximately 0.25 standard deviations (Hausknecht et al., 
2007). Retaking the ACT has shown a 1- to 2-point scale score increase (0.17 to 

0.34 SD unit) across testing experiences (Andrews & Ziomek, 1998). Furthermore, 
students who first tested as a junior and retested later had an average ACT Composite 
score gain of 1.1 points, i.e., 0.19 SD units (Harmston & Crouse, 2016). These gains 
are likely due to practice effects as well as growth from high school instruction and 
continued learning opportunities. 


Practice effects may actually work to bolster test validity. For example, when students 
are negatively impacted by test anxiety, their ability to demonstrate their content 
knowledge is impaired. Practice has the potential to alleviate the negative impact, 
thereby making a score more representative of domain mastery (Messick, 1982). We 
explore the relationship between practice opportunities and students’ ACT scores 

by modelling the relationship of self-reported test preparation activities and ACT 
Composite score, separately for first-time and repeat tested students. 


Socioeconomic Status. Test preparation for college entrance exams can vary in 

cost from free to thousands of dollars. As a result, it is no surprise that access to test 
preparation can vary by income, with some studies finding more affluent students 

more likely to participate in more expensive and more individualized test preparation 
programs (Buchmann, Condron, & Roscigno, 2010; Park & Becks, 2015). A recent 
study on paid users of ACT Online Prep (AOP), an online subscription-based test 
preparation program, found that lower- and middle-income students had greater odds 
of enrolling in the program than higher-income students (Sanchez, 2019). AOP is a less 
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expensive test preparation option compared to one-on-one and in-person classroom 
options. Buchmann, Condron, and Roscigno (2010) note that higher-income families 
are more likely to use more expensive and potentially more in-depth test preparation 
programs such as private courses and tutors relative to families with lower incomes, 
which make use of more economical test preparation programs such as workbooks, 
software, or videos. As such, it is important to examine the impact of different types of 
test preparation programs as well as how that effect differs by family income levels. We 
must be cognizant of the financial barriers that may preclude the participation of some 
students in certain types of test preparation. We, therefore, investigate whether the 
benefits of different types of test preparation activities on ACT scores varies by family 
income. 


Goals of the Study 


While there is some evidence that retesting (which includes practice opportunities) can 
improve test scores, there is limited research on whether individual factors, including 
students’ grade level and family income, play a role in the relationship between different 
test preparation activities and ACT scores. Therefore, in this study, we seek to answer 
these research questions: 


(RQ1) Is there a relationship between type of preparation activity and ACT 
Composite score? 


Given the abundance of research that indicates engaging in some type of test 
preparation improves test scores, we expect the same to hold true in this study. 
Further, while there is past evidence to support that practice testing improves ACT test 
scores relative to no test prep at all, less is known about whether and to what degree 
in-person and self-paced instruction improves test scores. We hypothesize that in- 
person instruction will improve ACT test scores relative to no test preparation, given 
the customized nature and hands-on learning in-person learning offers. Self-paced 
preparation is expected to also improve test scores but not to the degree of in-person 
preparation given the wide variety of self-paced approaches, many of which depend on 
students’ motivation in order to be used effectively. 


(RQ2) Does the relationship between type of preparation activity and ACT 
Composite score depend on students’ family income or whether students 
are first-time or repeat ACT testers? 


Family income. We are interested to see if students who come from lower-income 
families may benefit more from in-person test preparation activities than students from 
higher-income families. It may be possible that students from lower-income families 
are not getting the type of instruction that resonates with them or provides the level of 
support needed at school, and in-person tutoring can provide the instruction to help 
them excel. 
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Repeat testing. As mentioned earlier, taking the ACT multiple times can be viewed as 

a form of test preparation as it is possible that the added prior testing experience might 
prime students on content covered, reduce test anxiety, and increase test familiarity. With 
retesting, students’ ACT scores are expected to increase for at least three reasons: (a) 
Prior exposure to the test brings an increased level of familiarity and comfort with taking 
the test, (b) Effects of test preparation that occur between the two testing periods, and (c) 
Natural growth from learning and instruction that occurs between the two testing periods. 
In this study, we include high school GPA (HSGPA) as the measure of prior achievement 
in an attempt to isolate the effects of test preparation.* Because all students who retest 
may benefit from prior exposure to the test, this approach estimates practice effects with 
test preparation. 


Methods 


Sampling Strategy 


An online survey was administered to a stratified random sample of students (N = 
45,400) who had registered to take the ACT in February 2016. Students who self- 
reported as Asian, Hispanic, or African American were over-sampled. White students, 
those who chose two or more races, and those who did not provide race/ethnicity were 
under-sampled. All available American Indian/Alaska Native and Native Hawaiian/Other 
Pacific Islander students were included. Students who retested more than one time prior 
to the February test date were also over-sampled; first-time test-takers were under- 
sampled. This sampling method was utilized to ensure sufficient representation across 
all racial/ethnic groups and by number of times tested. Weights are used to analyze the 
data; more information on the weighting procedure will be discussed later in this section. 


Survey 


Email addresses for ACT test-takers were obtained from students’ registration record for 
their February 2016 ACT test date. This contact information was then used to send out 
an invitation to participate in a survey about their test preparation activity prior to taking 
the ACT. The invitation described the purpose of the study and indicated that participation 
was completely voluntary and would in no way affect students’ ACT scores. Survey data 
collection proceeded for two weeks. 


The survey was administered the same day as, and approximately three hours after, 

the February National ACT test administration. Two reminder emails were sent to the 
survey sample. Approximately 16% (n=7,640) of the sample responded to the survey. No 
incentives were provided for participation, and the survey took, on average, about five 
minutes to complete. 


The survey focused on three areas: student race/ethnicity (two questions), test 
preparation activities (nine questions), and factors influencing survey participation (two 
questions). In the present study, we focused on a question which asked “Which of the 
following test preparation activities, if any, did you use to prepare for the February 6th 
ACT test? (Choose all that apply.).” Table 1 provides the list of preparation activities and 
the percentage of students who chose that activity. 
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Table 1. Number and Percentage of Students by Test Preparation Activity 


Test Preparation Activity % 


Self-paced review of the content on the ACT using paper 


materials (e.g., workbooks) Zo oe 
ie review of the content on the ACT (excluding 967 12.7% 
Beale online review of the content on the ACT, using online 539 71% 
In-person small group instruction (less than six students) 449 5.9% 
In-person large group instruction (six or more students) 855 11.2% 
In-person one-on-one tutoring 900 11.8% 
Online small group instruction (less than six students) 54 0.7% 
Online large group Instruction (six or more students) 106 1.4% 
Online live one-on-one tutoring 25 1.2% 
Only practice assessments (e.g., practice tests, drills, or quizzes) 224 2.9% 
Other (please explain) 1,375 18.0% 
| have not participated in any test preparation activities 1,207 15.8% 


Note: Responses were able to chose more than one test preparation activity, with a few 
exceptions. Respondents who chose “only practice assessments” or “I have not participated in 
any test preparation activities” were unable to choose another test preparation activity. 


Due to the sparseness of responses and the overlapping nature of the test preparation 
activities provided, we collapsed responses into four categories: self-paced 
preparation, in-person preparation, other test preparation activities, and using no test 
preparation materials. Among respondents, only 1.5% said they used a form of online 
test preparation in small group, large group, or one-on-one formats. Furthermore, 
about 75% of students reported only participating in one of the following types of test 
preparation: in-person, self-paced, other, or no test preparation. We therefore focus on 
these students in our analysis. Of these 5,702 respondents, we also limited our sample 
to those who were not missing any of the variables used in the regression models, with 
the exception of family income, which was imputed. Given these constraints, we focus 
on 4,572 students in our analysis. 


Analytical Sample 


Because we administered the survey to a stratified random sample based on race and 
number of times tested, and because a disproportionate number of White students, 
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female students, and students with higher academic achievement participated in 

the survey, we weighted the survey responses. Weighting reduces the chances for 
non-response bias, corrects for the stratified sampling conducted and, subsequently, 
on key measures expected to relate to test preparation behavior, makes the survey 
respondents more representative of the 2016 February tested population to which 
they are being generalized. Propensity score weighting was employed. Here, a logistic 
regression model was estimated predicting the probability of survey participation, 
given population characteristics. We used students’ race/ethnicity, gender, ACT 
Composite score, parents’ education level, number of times tested, and income levels 
as predicators. Missing data were imputed prior to fitting the logistic regression model. 
The appendix provides descriptive statistics for the unweighted sample, weighted 
sample, and the population (see Appendix). 


In the analytical sample, we see that relatively fewer students came from households 
with a low family income, and most students came from a middle-income family 
(Table 2). There were slightly more female students than male students in the sample, 
and the majority of students were White. Most students had not taken advanced 
coursework in science or mathematics but had taken a college-prep curriculum in high 
school and indicated needing some academic or career navigation help. Just over 40% 
of students came from a household whose highest parental education was less than 
an associate degree, and most students expected to complete at least a bachelor’s 
degree. The February 2016 ACT test was the first test for half of the analytical sample, 
and over 60% of the sample were in the 11th grade. Finally, there were relatively 
similar numbers of students participating in each test preparation activity group. 


Table 2. Demographic characteristics for the analytical sample 


Be} r-1 MCT colt] e) 
Variable % (N) 
Family Income 
<$36,000 18.7 (934) 
$36,000-$100,000 30.0 (1,901) 
>$100,000 24.6 (1,231) 
Missing 26.8 (1,342) 
Gender 
Female 57.6 (2,884) 
Male 42.4 (2,124) 
Race/Ethnicity 
African American 16.5 (825) 
American Indian 3.4 (172) 
White 60.0 (3,005) 
Hispanic 13.2 (660) 
Asian 4.7 (235) 
Native Hawaiian 0.9 (47) 
Two or More Races 0.7 (35) 


Prefer Not/No Response 0.6 (31) 
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Table 2. Demographic characteristics for the analytical sample—continued 


Variable 
Take Bio, Chem, Physics 
No 
Yes 
Taken Math Beyond Algebra Il 
No 
Yes 
Take a College Curriculum 
No 
Yes 
Need Help in academic or career navigation* 
No 
Yes 
Highest Parental Education 
Less than Associates 
Bachelor's degree 
Graduate degree 
Missing 
Expected Educational Attainment 
Associate Degree/Voc-Tech 
Bachelor’s Degree 
1-2 Yrs. of Grad Program/Professional Degree 
Other/No Response 
Number of Times Tested 


Test Preparation Activity 
In-person Instruction 
No Prep 
Other 
Self-Paced 

Grade Level 
11 
d2Z 


BKelr-|MeTcolt]e) 


% (N) 


64.1 (3,210) 
35.9 (1,798) 


64.1 (3,210) 
57.7 (2,888) 


37.4 (1,874) 
62.6 (3,134) 


34.5 (1,729) 
65.5 (3,279) 


42.6 (2,135) 
29.3 (1,470) 
28.0 (1,404) 


3.5 (176) 
45.7 (2,287) 
41.8 (2,094) 

9.0 (452) 


50.4 (2,525) 
37.6 (1,884) 
6.9 (344) 
2.9 (147) 
2.2 (108) 
21.4 
27.4 


23.8 
2° 


1,069) 
1,374) 
1,190) 
1,375) 


a nn on. 


64.4 (3,227) 
35.6 (1,781) 


Note. *This included students’ indications of need for help with deciding educational and 


occupational plans, expressing ideas in writing, improving reading speed and comprehension, 


study skills, or mathematical skills. 


ACT Research & Policy | Technical Brief | November 2020 8 


Table 3 summarizes characteristics of the weighted analytic sample by type of test 
preparation.’ A greater percentage of students from higher-income families, as well as 
students who did not report family income, were more likely to report using in-person 
instruction in comparison to other income groups. On the other hand, more African 
American students reported using self-paced or other test preparation; similar numbers 
of Hispanic students reported using self-paced, other, or no test preparation; and 

more White students reported using self-paced or no test preparation. When we look 
at in-person test preparation, we see that more White students used this type of test 
preparation (23.3%) than African American students (20.4%), and both used in-person 
test preparation more than Hispanic students (17.2%). We can also see that fewer first- 
time test-takers use in-person test preparation than retesting students. 


Table 3. Demographic Characteristics for the Weighted Sample 


In-person 


: Y=1 i ers Kexe\e| 
Instruction 


WET SE 10) (-) 
Family Income 


% (N) 


% (N) 


<$36,000 Ted (128) 3131 (291) 27.7 (259) Zi. (257) 
$36,000—-$100,000 13.3 (199) 30.9 (463) 24.3 (364) 31.6 (475) 
>$100,000 28.3 (348) 25.6 (315) 19.2 (237) 26.9 (332) 
Missing 29.4 (395) 22.8 (306) 24.6 (330) 23:1 (310) 
Gender 
Female 19.5 (563) 28.2 (813) 24.8 (716) 27.5 (793) 
Male 23.8 (506) 26.5 (562) 22.3 (475) 27.4 (581) 
Race/Ethnicity 
African American 20.4 (168) 30.6 (252) 26.0 (214) 23.1 (191) 
American Indian 14.9 (26) 27.2 (47) 23.3 (40) 34.6 (59) 
White 23.3 (698) 26.2 (786) 21.3 (640) 29.3 (880) 
Hispanic 17.2 (113) 26.7 (176) 28.2 (186) 27.9 (184) 
Asian 18.2 (43) 33.6 (79) 31.4 (74) 16.9 (40) 
Native Hawaiian 21.4 (10) 24.7 (12) 40.0 (19) 13.9 (6) 
Two or More Races 14.5 (5) 40.0 (14) 18.7 (7) 26.8 (9) 
Prefer Not/No Response' 20.9 (6) 28.9 (9) 35.4 (11) 14.8 (5) 
Take Bio, Chem, Physics 
No 19.7 (631) 28.2 (905) 23.5 (754) 28.7 (920) 
Yes 24.4 (439) 26.1 (470) 24.3 (436) 25.2 (453) 
Taken Math Beyond Algebra II 
No 19.7 (631) 28.2 (905) 23.5 (754) 28.7 (920) 
Yes 21.4 (619) 27.3 (789) 24.3 (700) 27.0 (780) 
Take a College Preparatory 
Curriculum 
No 16.7 (314) 30.0 (562) 25.4 (475) 2179 (923) 
Yes 24.1 (756) 25.9 (813) 22.8 (715) Zi hoo) 
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Table 3. Demographic Characteristics for the Weighted Sample—continued 


EVs Le) (<1 

Need help in academic or 

career navigation 
No 
Yes 

Highest Parental Education 
Less than Associates 
Bachelor's degree 
Graduate degree 


Expected Educational 
Attainment 


Associate Degree/Voc-Tech 
Bachelor’s Degree 


1-2 Yrs. of Grad Program/ 
Professional Degree 


Other/No Response 
Number of Times Tested 

1 

2 


5+ 
Grade Level 

11 

12 


In-person 


Instruction 


25.5 (442) 
19.1 (628) 


17.0 (362) 
22.1 (325) 
27.2 (382) 


22.3 (39) 
19.5 (446) 
21.3 (446) 


30.6 (138) 


17.7 (446) 
25.3 (477) 
23.3 (80) 
27.6 (41) 
23.2 (25) 


25.0 (806) 
14.8 (263) 


Self-Paced 


% (N) 


26.5 (459) 
27.9 (916) 


27.7 (590) 
25.6 (376) 
29.1 (408) 


38.1 (67) 
28.7 (657) 
27.2 (570) 


17.8 (80) 


26.5 (670) 
28.6 (539) 
28.7 (99) 
26.3 (39) 
27.0 (29) 


24.5 (791) 
32.8 (584) 


Other 
% (N) 


21.1 (366) 
25.2 (825) 


28.2 (601) 
21.4 (314) 
19.6 (275) 


21.6 (38) 
23.4 (535) 
24.1 (505) 


24.8 (112) 


25.0 (632) 
22.0 (414) 
23.1 (79) 
27.7 (41) 
22.5 (24) 


23.6 (761) 
24.1 (430) 


Note. *This included student’s indication of a need for help with deciding educational and 


occupational plans, expressing ideas in writing, improving reading speed and comprehension, 


study skills, or mathematical skills. 


% (N) 


26.8 (463) 
27.8 (911) 


27.2 (581) 
30.9 (454) 
24.1 (339) 


18.0 (32) 
28.4 (648) 
27.4 (573) 


26.8 (121) 


30.8 (778) 
24.1 (453) 
25.0 (86) 
18.5 (27) 
27.3 (29) 


27.0 (870) 
28.3 (504) 
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Measures 


Test Preparation Activity. Type of test preparation was based on responses to the 
survey item previously described. Responses were grouped as in-person instruction, 
self-paced test preparation, other test preparation, and not having used test 
preparation. 


Student Profile Section of the ACT Registration. As part of the registration process, 
students self-report gender, race/ethnicity, grade level, family income, expected 
educational attainment, and highest parental education. Race/ethnicity was categorized 
as African American, American Indian/Alaska Native, Hispanic/Latino, Asian, Native 
Hawaiian/Other Pacific Islander, two or more races, or chose not to respond. For grade 
level, students were asked what grade they are currently in and were given options 
from 7th grade to 12th grade. Only students in 11th and 12th grade were used in this 
study. For family income, students are asked “please estimate the approximate total 
combined income of your parents before taxes last year.” While family income ranges 
include nine categories from “less than 24,000” to “more than $150,000,” for analysis 
purposes we collapsed the categories to three: less than $36,000, between $36,000 
and $100,000, and greater than $100,000. Students are also asked “what is the 
highest level of education you expect to complete?” While this question asked about 
six educational levels, for analysis, we collapsed the categories into four: associate 
degree/voc-tech, bachelor’s degree, 1-2 years of a graduate program/professional 
degree, and other/no response. This section also asks “what is the educational level 

of your mother/guardian 1 (or father/guardian 2)?” This section provides eight ordered 
response options. For analytical purposes, we collapsed these to associate degree or 
less, bachelor’s degree, or graduate degree or higher, and we used highest parental 
education for either parent/guardian. 


In addition to this demographic information, students are asked about their academic/ 
career needs and interests. In our study, we made use of students’ self-reported 
need for assistance in five specific areas. These five areas included “deciding my 
educational and occupational plans,” “expressing my ideas in writing,” “improving my 
reading speed and comprehension,” “i 


improving my study skills,” and “improving my 
mathematics skills.” In the current study, if a student responded affirmatively to any of 
the five items, they were classified as having a self-identified need for support. 


Course Grade Information Section of the ACT Registration. In this section of the 
ACT Registration, students are asked about the specific courses they either have taken 
or plan to take. They are also asked to provide grades for each of the courses they 
have taken up to the point of registering for the ACT. HSGPA, as used in this study, is 
based on students’ self-report of grades from up to 23 high school courses in English, 
mathematics, social studies, and natural science. Additionally, we used course-taking 
information to identify students’ mathematics and science coursework patterns. We 
identify whether students had taken mathematics coursework beyond Algebra II and 
if students had taken at least Biology, Chemistry, and Physics. Finally, students were 
asked to indicate whether they were taking a college preparatory curriculum in high 
school. 
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Number of Times Tested. ACT test records were reviewed for all instances of a 
student taking the ACT exam on or before February 2016, the national test date 
students were asked to report their test preparation approach about. The number of 
times a student had taken the ACT was calculated from these data. 


Analysis 


Income was a focus in our analyses, and since this variable was missing for 29% of 
cases, we utilized multiple imputation for missing values (Rubin, 1987). Here, imputed 
values replace the missing values to create a dataset with no missing data on family 
annual income. Imputation is conducted multiple times, in this case five times, to create 
multiple data sets that vary on the imputed income. The imputed values were estimated 
using variables that were correlated with ACT Composite score. The multiple imputation 
model included students’ expected education level; the number of years of English, 
mathematics, social science, and natural science coursework; HSGPA; gender; race/ 
ethnicity; family income; and highest parental education. It is worth noting that the 
intent of multiple imputation is not to guess an individual respondent’s data point; 
rather, the intent is to analyze data that maintains the variability and relationship of all 
the variables in the model. 


Linear regression was used to model the relationship between different types of test 
preparation and ACT Composite score, controlling for academic and demographic 
factors (RQ1). To evaluate the interaction between types of test preparation activities 
and family income on ACT Composite score, the model was fit separately for lower-, 
middle-, and higher-income students (RQ2). Then, the model was run separately for 
first-time and repeat testers to estimate the interactive effects of test preparation activity 
type and number of times tested on ACT Composite score (RQ2). As a reminder, we 
refrained from adding students’ prior ACT scores as covariates when modeling repeat 
test-takers. Instead, we used HSGPA as a control variate for students’ academic 
achievement. Weights were used in all regression models. 


12 
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Table 4. Model Rregression Ccoefficients Predicting ACT Composite Score 


First-time 
Total Group Low-Income Middle-Income —__High-Income Testers Repeat Testers 

Predictor Std Err Std Err Std Err Std Err StdErr Est StdErr 
Intercept 9.3 0.5 12.9 1.0 8.5 1.1 6.0 1.6 9.0 0.7 11.1 0.8 
HSGPA 3.3 0.1 2.2 0.2 3.6 0.2 4.0 0.3 3.4 0.2 3.1 0.2 
Race Ethnicity 

African American -2.4 0.2 -3.0 0.2 -2.5 0.3 -1.8 0.5 -2.5 0.3 -2.2 0.2 

American Indian -0.7 0.3 -1.4 0.4 -0.6 0.5 -0.4 0.7 -0.8 0.4 -0.4 0.4 

White 0.0 0.0 

Hispanic -1.3 0.2 -2.3 0.3 -1.3 0.3 -0.1 0.5 -1.4 0.3 -1.3 0.2 

Asian 0.9 0.3 -0.4 0.5 1.0 0.5 1.6 0.5 0.9 0.4 0.8 0.4 

Native Hawaiian 0.8 0.6 0:2 1.4 -0.4 0.9 2.4 1.0 0.8 0.7 0.6 41:2 

Two or More Races 2.1 0.6 0.9 1.4 2.4 1.0 1.7 1:3 0.7 1.4 2.6 0.7 

Prefer Not to Respond 1.4 0.7 -0.4 5:3 -0.2 22 2:3 1.9 1.3 1.4 1.4 0.8 
Family Income 

< $36,000 -0.4 OR - - - - - - -0.2 OFS -0.6 0.2 

$36,000-$100,000 = 5 = = 5 e 

> $100,000 0.3 0.1 - - - - - - 0.5 0.2 0.2 0.2 
Gender 

Female -0.8 0.1 -0.8 0.2 -0.8 0.2 -0.8 0.2 -0.7 0.2 -0.9 0.2 
Student Expected Education 
Attainment 

Associates Degree/Voc-Tech 

Bachelor’s Degree 0.7 0.3 0.8 0.4 0.3 0.5 leo 1.4 ORT 0.4 0.4 O25) 

1-2 Yrs. of Grad Program/ 1.7 0.3 2.1 0.4 ile ORS 215) 1.4 1.7 0.4 1.4 0.5 

Professional Degree 

Other/No Response 0.2 0.4 0.0 0.6 -0.2 0.8 |Z ANd 0.6 0.5 -0.5 0.6 


Highest Parental Education 
Less than Associates 
Bachelors 0.0 0.2 0.2 0.3 -0.2 0.2 0.3 0.4 0.0 0.2 0.1 0.2 
Graduate 1.3 0.2 i a 0.5 1.4 0.3 1:3 0.4 1.1 0.3 1.4 0.2 
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Table 4. Model Rregression Ccoefficients Predicting ACT Composite Score—continued 


First-time 


Total Group Low-Income Middle-Income —__High-Income Testers Repeat Testers 


ea k:Xe| (eae) g Est StdErr_ Est Std Err Est Std Err Est StdErr Est StdErr Est StdErr 


Need Help with Educational or 
Occupation Planning 


Yes -1.4 0.1 -1.3 0.3 -1.3 0.2 -1.4 0.2 -1.4 0.2 -1.3 0.2 
Taking a College Curriculum 

Yes 0.9 0.1 0.5 0.2 1.0 0.2 1.1 0.3 1.0 0.2 0.7 0.2 
Taken Mathematic Beyond Algebra II 

Yes 1.6 0.1 1.3 0.2 1.7 0.2 1.6 0.2 1.5 0.2 1.7 0.2 
Taken Biology, Chemistry, & Physics 

Yes 1.0 0.1 0.8 0.3 1.0 0.2 1.2 0.2 0.9 0.2 1.2 0.2 
Type of Test Preparation 

In-person instruction 0.1 0.2 -0.4 0.3 0.1 0.3 0.5 0.3 0.3 0.2 0.0 0.2 

Self-paced -0.2 0.1 -0.5 0.3 -0.2 0.3 -0.1 0.3 -0.5 0.2 0.0 0.2 

Other -0.1 0.2 -0.3 0.3 -0.3 0.3 0.0 0.3 -0.1 0.2 -0.1 0.2 

No prep 
Grade Level 

12 -1.7 0.1 -1.7 0.2 -1.7 0.2 -1.7 0.3 -1.3 0.2 -2.0 0.2 
Number of Times Tested 0.3 0.1 0.4 0.1 0.5 0.1 0.1 0.1 - - - - 


Note: Values in the table in bold font indicate significant at the 0.05 level. 
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Results 


(RQ1) Is there a relationship between type of preparation activity and ACT 
Composite score? 


As shown in Table 4, the regression model for the total group found that HSGPA; race/ 
ethnicity; gender; students’ expected educational attainment; highest parental education; 
needing help with educational or career planning; taking a college-prep curriculum; 
taking mathematics beyond Algebra II; taking at least Biology, Chemistry, and Physics; 
grade level; and the number of times taking the ACT were all significant predictors of 
student performance on the ACT. For the total group, ACT Composite score did not vary 
by type of test preparation. 


(RQ2) Does the relationship between type of preparation activity and ACT 
Composite score depend on students’ family income or whether students 
are first-time or repeat ACT testers? 


Looking first at students from low-income families, type of test preparation was not 

a significant predictor of ACT Composite score (see Table 4). The same was true for 
students from both middle- and high-income families. This suggests that the effect of 
test preparation does not differ between family income groups.* 


Assessing whether the effects of test preparation varied for first-time and repeat testers, 
we see that first-time test-takers who participated in self-paced test preparation had a 
lower ACT Composite score than those first-time test-takers who did not participate in 

a test preparation activity (see Table 4). There was no predictive power of preparation 
activities on ACT Composite score for those who took the test more than once. 


Discussion 


Because the ACT test can have important consequences for postsecondary access, 
students often elect to participate in a wide variety of test preparation activities. In the 
current study, we focused on how three different test preparation formats—in-person 
instruction, self-paced, and other types—related to students’ ACT Composite scores. 
Furthermore, we investigated whether the effects of these types of test preparation 
differed by family income and repeat testing status. 


We found that in the income subgroup models, ACT Composite score did not vary by 
test preparation activity. This is interesting to note because there is a concern that 
higher-income families may be afforded greater opportunities for more impactful test 
preparation. The present study did not suggest that to be the case. Notwithstanding this 
finding, this is an area that deserves consideration because we know that longer-term 
learning opportunities that parallel in-class instruction can have an impact on student’s 
scores by bolstering core content knowledge (Appelrouth & Zabrucky, 2015; ACT, 2005). 
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If test preparation mirrors in-depth learning opportunities, there is a greater opportunity 
for both short-term benefits in the form of increases in standardized test scores as well 
as long-term benefits in content mastery and coursework performance. These types 

of in-depth learning opportunities are typically more expensive and less utilized by 
lower-income families (Buchmann, Condron, & Roscigno, 2010; Park & Becks, 2015). 
Previous research on test preparation shows a connection between the use of test 
preparation and an increase in ACT scores. It would be worthwhile to further investigate 
whether the use of test preparation is also accompanied by longer-term content 
mastery as demonstrated by high school course performance. 


In our study, we examined retest effects (e.g., benefit of taking the ACT at least once 
prior to current test) by modeling the effect of number of prior ACT tests. We found that 
number of times tested had an impact on ACT scores and the effect of test preparation 
format differed for first-time and repeat test-takers. 


This study is not without its limitations. We examined general categories of test 
preparation, and examination of test preparation activities as in-person, self-paced, or 
other incorporates many types of activities under each of those three categories. Types 
of activities in any one category can vary by intensity, duration, quality, content covered, 
mode of delivery and pacing, among other attributes. This provides for an estimate 

of the average effect for these classes of test preparation; however, there is probably 
great variability in effects within each group. The time allocation and pacing of test prep 
has important implication for the potential efficacy of any program. It is possible that 

the aggregation conducted in this study is masking true underlying effects for specific 
subtypes of activities or for specific, particularly effective, programs. It is important to 
operationalize test prep in a way that captures the many considerations in self-selecting 
to use test preparation, how students use test preparation, and considers all the types 
of preparation students use. 


Moreover, we utilized an “other” category of test preparation. Specifically, this included 
1,375 students who had some type of unidentified test preparation. One reason so 
many students classified their test preparation as “other” is that the descriptions of 

the types of test preparation may not have been sufficiently interpretable to allow 
students to identify as having participated in that type of test preparation. Another 
possible reason is that there may be some types of test preparation we did not capture. 
Both possibilities present analytical challenges. In the first case, if students with the 
same type of test preparation self-identify in different types of test preparation, it is 
more difficult to isolate the effect of that type of test preparation. The second possible 
reason means students who self-identified as other may have been more appropriately 
classified in the named test preparation activities. It is also possible that if they were 
accurately classified the effect for that named group would have been more precise. 
This presents an interesting perspective into the understanding of test preparation as 
well. These large numbers of “other” respondents present a learning opportunity to 
better understand common test preparation efforts. 


Finally, given the cross-sectional design and the self-selection of students into the 
test preparation activities, we cannot make causal statements about test preparation 
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and ACT performance. The results of this study, based on traditional regression 
techniques, should not be overinterpreted to say that test preparation has no effect on 
ACT Composite scores. Rather, what we can say from the present study is that when 
we aggregate self-reported test preparation into three broad types of test preparation, 
i.e. in-person instruction, other test preparation, and self-paced test preparation, an 
association between test-preparation and ACT Composite scores was not found. 


Our research does not align with past research that shows a small effect of test 
preparation on ACT scores. Our study might not have detected an effect because of 
the research design or the sample of students who chose to participate in the survey. 
Likewise, when we aggregate types of test preparation into very broad categories, we 
may be losing the ability to find efficacy of specific types of test preparation activities 
and programs. Therefore, specific test preparation programs and activities need to be 
evaluated to identify high efficacy activities and programs. Students were instructed to 
choose all test preparation activities that they engaged in, and, accordingly, students 
were not in mutually exclusive test preparation types. Utilizing a model with all test 
preparation activities together helps isolate the effects of multiple, simultaneously 
engaged in, test preparation activities. Collapsing the test preparation activities as 
we had allowed us to put students into one activity and run a parsimonious number of 
models. 


Notes 


1. On national 2018-2019 ACT test dates, students were asked “Did you prepare for 
the ACT® test using any test preparation materials (for example, The Official ACT 
Prep Guide, other study guides, online materials, practice tests, tutors, or test prep 
courses)?” This percentage is based on students who responded to the question. 


2. It would also be possible to utilize a prior ACT Composite score to account for 
prior achievement. If that prior achievement measure were used, however, it would 
necessitate the use of repeat tested students only. 


3. Characteristics for the national February 2016 ACT test-taking population are 
presented in the Appendix. 


4. An alternative analytical approach to conducting separate models for each level of 
family income would be to run a single model with an interaction between family 
income and type of test preparation. To evaluate if model choice would result in 
different conclusions, we ran the interaction model and found that the interaction 
was not significant at the .05 level. 
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Appendix 


Table A1. Unweighted, Weighted, and Total Population Demographics—continued 


MYETat-le) (1 


Family Income 
<$36,000 
$36,000-$100,000 
>$100,000 
Missing 
Gender 
Female 
Male 
Race/Ethnicity 
African American 
American Indian 
White 
Hispanic 
Asian 
Native Hawaiian 
Two or More Races 
Prefer Not/No Response' 
Take Bio, Chem, Physics 
No 
Yes 
Missing 
Taken Math Beyond Algebra Il 
No 
Yes 
Missing 
Take a College Curriculum 
No 
Yes 


Need Help in academic or career 
navigation 


No 
Yes 

Highest Parental Education 
Less than Associate 
Bachelor's Degree 
Graduate Degree 
Missing 


Unweighted 


Sample 
% (N) 


23.2 
20:2 
20.9 
27.4 


1,159) 
1,414) 
1,046) 
1,388) 


a a ae 


65.2 (3,262) 
34.9 (1,745) 


15.4 (769) 
4.1 (205) 
26.8 (1,344) 
19.9 (997) 
31.3 (1,566) 
1.1 (57) 
0.7 (36) 
0.7 (33) 


57.1 (2,861) 
42.8 (2,144) 
0.0 (2) 


57.1 (2,861) 
64.1 (3,210) 
0.0 (2) 


38.2 (1,913) 
61.8 (3,094) 


33.4 (1,672) 
66.6 (3,335) 


44.8 (2,241) 
25.7 (1,286) 
29.6 (1,480) 


Weighted 
Sample 
% (N) 


18.7 (934) 
30.0 (1,501) 
24.6 (1,231) 
26.8 (1,342) 


57.6 (2,884) 
42.4 (2,124) 


16.5 (825) 
3.4 (172) 
60.0 (3,005) 
13.2 (660) 
4.7 (235) 
0.9 (47) 
0.7 (35) 
0.6 (31) 


64.1 (3,210) 
35.9 (1,798) 
0.0 (0) 


64.1 (3,210) 
57.7 (2,888) 
0.0 (0) 


37.4 (1,874) 
62.6 (3,134) 


34.5 (1,729) 
65.5 (3,279) 


42.6 (2,135) 
29.3 (1,470) 
28.0 (1,404) 


February 
2016 Tested 
reXe) LUI E-ViCey a] 
% (N) 


18.0 
29.6 
23,8 
28.0 


58,930) 
93,377) 
75,017) 
88,550) 


a a a 


55.2 (174,332) 
44.8 (141,542) 


15.7 (49,718) 
0.7 (2,249) 
58.1 (183,472) 
12.5 (39,343) 
4.6 (14,595) 
0.2 (589) 
3.8 (11,908) 
4.4 (14,000) 


64.8 (204,636) 
35.2 (111,034) 
0.1 (204) 


64.8 (204,636) 
57.2 (180,778) 
0.1 (203) 


37.1 (117,051) 
62.9 (198,823) 


36.4 (115,060) 
63.6 (200,814) 


33.6 (53,103) 
25.6 (106,035) 
24.0 (80,966) 
16.8 (53,103) 
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Table A1. Unweighted, Weighted, and Total Population Demographics—continued 


MYETat-le) (1 


Expected Educational Attainment 
Associate Degree/Voc-Tech 


Bachelor’s Degree 


1-2 Yrs of Grad Program/ 


Professional Degree 
Other/No Response 
Number of Times Tested 
1 
2 


5+ 
Test Preparation Activity 
In-person Instruction 
No Prep 
Other 
Self-Paced 
Grade Level 
a 
12 


ACT Composite score, mean 


HSGPA, mean 


Unweighted 


Sample 
% (N) 


2.9 (144) 
38.0 (1,902) 
49.5 (2,478) 


9.7 (483) 


52.7 (2,640) 
37.5 (1,879) 
5.7 (284) 
2.6 (128) 
1.5 (76) 


18.9 (946) 

24.1 (1,207) 
27.5 (1,375) 
29.5 (1,479) 


68.2 (3,414) 
31.8 (1,593) 
23.1 
3.5 


Weighted 
Sample 
% (N) 


3.5 (176) 
45.7 (2,287) 
41.8 (2,094) 


9.0 (452) 


50.4 (2,525) 
37.6 (1,884) 
6.9 (344) 
2.9 (147) 
2.2 (108) 


21.4 
27.4 
23.8 
Zia 


1,069) 
1,374) 
1,190) 
1,375) 


—a~ a ~~ 


64.4 (3,227) 
35.6 (1,781) 
247 
3.4 


February 
2016 Tested 
ree) oLUIF-Licey a 
% (N) 


3.2 (10,234) 
46.3 (146,247) 
40.9 (129,224) 


9.6 (30,169) 


54.6 (172,596) 
26.8 (84,497) 
10.7 (33,692) 
4.5 (14,047) 

3.5 (11,042) 


59.5 (188,057) 
40.5 (127,817) 
210 
3.4 
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